Pneumonia Detector from X-ray Images

Background:

Chest radiography (X-ray) remains the most common radiological exam in the world and done as a first screening tool for a wide range of conditions (Raoof et al., 2012). In 2006, it is estimated that 129 million CXR images were acquired in the United States alone. The demand comes from low-cost and low radiation dose, and reasonable sensitivity to a wide variety of pathologies.

The interpretation of the chest radiograph can be challenging due to the superimposition of anatomical structures along the projection direction. This effect can make it very difficult to detect abnormalities in particular locations. This causes radiologists to show high inter-observer variability in their analysis of the images (Balabanova et al., 2005).

The volume of images acquired, the complexity of their interpretation, and their value in clinical practice have long motivated researchers to build automated algorithms for analysis. The trend in using modern machine learning techniques started in 2017 when NIH released 112,000 images from NIH clinical centers. And in 2019 more than 750,000 additional images were released in 3 labeled datasets.

The potential gains from ML-based automated analysis include increased sensitivity for subtle findings, prioritization of time-sensitive cases, automation of tedious daily tasks, and provision of analysis in situations where radiologists are not available. In addition, such tools can also be a first step before an experienced doctors reviews the result. It would increase confidence and reduce overall false positive or false negative diagnosis.

Problem Statement:

The goal is to reliably predict the probability of Pneumonia in a patient from the chest x-ray. While many papers are published in this field. It seems the focus has been on architecture of CNN that can achieve the classification. While in medical field an accurate estimate of the probability is critical when making diagnosis.

Solution Statement:

We intend to train a CNN model to classify the X-ray images into Normal and Pneumonia. For more efficient and faster training we load weights from a pre-trained ResNet model. A softmax at the output of the last layer gives us the uncalibrated probabilities of the Pneumonia vs. Normal.

We train another model using beta calibration model to go from uncalibrated probabilities into calibrated probabilities. The concept for calibration is the if for example the probability of Pneumonia is $p \%$ for some patients, we expect that actually $p \%$ of such patients have Pneumonia. This is a much more useful information to the physician that need to make decisions for the patient.

To calibrate this second model, we first need to split our training dataset into train and calibration sets. The calibration set does not need to be large as there are not so many parameters to tune.

After the calibration model is trained/fitted. The outputs of the CNN is softmaxed and passed to the calibration model to determine the probability of the Pneumonia from the patient's X-ray.

Datasets:

There are several chest X-ray datasets available from different sources. We downloaded this Dataset from Kaggle. The Original source of data is University of California San Diego in here

The dataset consist of 5840 total 2D Black and White X-ray images. The dimension of the images varies in the dataset and therefore a consistent resizing is needed. The dimensions are in the range of (900 to 1400) x (600 to 1300). When doing resizing we should take note of possible image aliasing and apply necessary smoothing filters if needed.

The labels are binary as NORMAL and PNEUMONIA. The train dataset has 3875/5216 images of PNEUMONIA and 1341/5216 of NORMAL. The test dataset 390/624 PNEUMONIA and 234/624 NORMAL images.

From the train dataset we pick 100 random images from Normal and 100 random images from Pneumonia and use them only for training the calibration model.

Benchmark Model:

For accuracy, precision, recall the model would be compared to: Kermany et. al.30154-5)

The benchmark for probability calibration is the dataset itself through a reliability diagram on the test dataset.

Evaluation Metrics:

  • a. Accuracy, Precision, Recall, F1-score :

    The overall accuracy of model should be high for the probability calibration to work properly. In this application Precision and Recall are both important as both False Positives and False Negative can impact the medical procedures that would be applied or not applied to a patient. Therefore, a combined F1-score can also be a suitable metric.

  • b. Reliability Diagram :

    We will look at probability diagram over the entire dataset. Make predictions based on calibrated probabilities and discuss the results.

Install and import on the instance

In [6]:
%%capture
import sys
!{sys.executable} -m pip install smdebug torch torchvision tqdm opendatasets betacal ml-insights==0.1.8
In [3]:
import ml_insights
ml_insights.__version__
Out[3]:
'0.1.8'
In [56]:
import os
import boto3
import sagemaker
from sagemaker.tuner import CategoricalParameter, ContinuousParameter, HyperparameterTuner
from sagemaker.pytorch import PyTorch
from sagemaker import get_execution_role
from sagemaker.debugger import Rule, DebuggerHookConfig, TensorBoardOutputConfig, CollectionConfig, ProfilerRule, rule_configs, ProfilerConfig, FrameworkProfile
from sagemaker.analytics import HyperparameterTuningJobAnalytics
import opendatasets as op
import numpy as np
import random

Get the data and copy it to S3

In [ ]:
%%capture
!pip install -U sagemaker

Download the data into notebook instance

In [6]:
op.download('https://www.kaggle.com/datasets/tolgadincer/labeled-chest-xray-images')
Downloading labeled-chest-xray-images.zip to ./labeled-chest-xray-images
100%|██████████| 1.17G/1.17G [00:10<00:00, 120MB/s] 

Copy the data into S3 bucket...

In [5]:
session = sagemaker.Session()
bucket = session.default_bucket()
region = session.boto_region_name
role = sagemaker.get_execution_role()

os.environ["DEFAULT_S3_BUCKET"] = bucket
In [9]:
%%capture

!aws s3 sync ./labeled-chest-xray-images s3://${DEFAULT_S3_BUCKET}/chestCRXImages/
In [6]:
print(bucket)

s3bucket = os.path.join("s3://", bucket)
print(s3bucket)

s3datapath = os.path.join(s3bucket, "chestCRXImages/chest_xray")
print(s3datapath)
sagemaker-us-east-1-286375333242
s3://sagemaker-us-east-1-286375333242
s3://sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray

Make a list of all the NORMAL and PNEUMONIA images in the train dataset. This is used to split the train dataset randomly into train and calibration...

In [7]:
s3 = boto3.resource('s3')
my_bucket = s3.Bucket(bucket)
NORMAL_images = []

for object_summary in my_bucket.objects.filter(Prefix="chestCRXImages/chest_xray/train/NORMAL"):
    #print(object_summary.key)
    NORMAL_images.append(object_summary.key)
    

PNEUMONIA_images = []
for object_summary in my_bucket.objects.filter(Prefix="chestCRXImages/chest_xray/train/PNEUMONIA"):
    #print(object_summary.key)
    PNEUMONIA_images.append(object_summary.key)
    
In [8]:
print(len(NORMAL_images))
print(len(PNEUMONIA_images))
1349
3883
In [12]:
Normal_len = len(NORMAL_images)
Pneumonia_len = len(PNEUMONIA_images)

Normal_index = random.sample(range(Normal_len), 100)
Pneumonia_index = random.sample(range(Pneumonia_len), 100)

Normal_images_calibration = [NORMAL_images[index] for index in Normal_index]
Pneumonia_images_calibration = [PNEUMONIA_images[index] for index in Pneumonia_index]

Calibration_images = Normal_images_calibration + Pneumonia_images_calibration

print("Moving some random files into Calibration folder...")
print("Bucker is " + bucket)
target_str = 'train/'
for image_name in Calibration_images:
    s_pos = image_name.index(target_str)
    new_image_name = image_name[:s_pos] + "Calibration/" + image_name[s_pos + len(target_str):]
    source_loc = bucket + '/' + image_name
    print('----------------------------')
    print("Moving " + source_loc + " to " + new_image_name)
    s3.Object(bucket, new_image_name).copy_from(CopySource=source_loc);
    print("Deleting " + source_loc)
    s3.Object(bucket, source_loc).delete();
Moving some random files into Calibration folder...
Bucker is sagemaker-us-east-1-286375333242
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-824917-0003.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-824917-0003.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-824917-0003.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-7298141-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7298141-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-7298141-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-4969331-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4969331-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-4969331-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8181293-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8181293-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8181293-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8272933-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8272933-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8272933-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-4507677-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4507677-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-4507677-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-4190635-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4190635-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-4190635-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-3852346-0002.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3852346-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-3852346-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8412621-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8412621-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8412621-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-6973172-0002.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6973172-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-6973172-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-3747940-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3747940-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-3747940-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-4271375-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4271375-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-4271375-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-9742274-0002.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9742274-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-9742274-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8644314-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8644314-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8644314-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-9084382-0003.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9084382-0003.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-9084382-0003.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-6081265-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6081265-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-6081265-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8229007-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8229007-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8229007-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-1522956-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1522956-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-1522956-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-5708209-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5708209-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-5708209-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-5918516-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5918516-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-5918516-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-6901859-0004.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6901859-0004.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-6901859-0004.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-2333546-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-2333546-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-2333546-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-7103127-0003.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7103127-0003.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-7103127-0003.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-263932-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-263932-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-263932-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8799208-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8799208-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8799208-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-3997421-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3997421-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-3997421-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-6044251-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6044251-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-6044251-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8688367-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8688367-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8688367-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8903542-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8903542-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8903542-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-9697349-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9697349-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-9697349-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-202916-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-202916-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-202916-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-7696091-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7696091-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-7696091-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-974509-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-974509-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-974509-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-5176049-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5176049-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-5176049-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-1044645-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1044645-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-1044645-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-3175613-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3175613-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-3175613-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-2048417-0002.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-2048417-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-2048417-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-5759621-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5759621-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-5759621-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-2387301-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-2387301-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-2387301-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-1830070-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1830070-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-1830070-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-7001302-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7001302-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-7001302-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-1128157-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1128157-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-1128157-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-2239132-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-2239132-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-2239132-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-9033279-0002.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9033279-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-9033279-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-3487615-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3487615-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-3487615-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-7436212-0003.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7436212-0003.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-7436212-0003.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8439083-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8439083-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8439083-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-9430515-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9430515-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-9430515-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-6915237-0001.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6915237-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-6915237-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8998369-0002.jpeg to chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8998369-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/NORMAL/NORMAL-8998369-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-5150733-0004.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5150733-0004.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-5150733-0004.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-5503803-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5503803-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-5503803-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-5882850-0002.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5882850-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-5882850-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-9566232-0002.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-9566232-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-9566232-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-5724853-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5724853-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-5724853-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-324273-0008.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-324273-0008.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-324273-0008.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-6065052-0002.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6065052-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-6065052-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-8313249-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8313249-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-8313249-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-9269035-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-9269035-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-9269035-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-8034950-0012.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8034950-0012.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-8034950-0012.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-9487238-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-9487238-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-9487238-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-1612812-0003.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-1612812-0003.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-1612812-0003.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-1191925-0002.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-1191925-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-1191925-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-4057077-0002.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-4057077-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-4057077-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-3371316-0005.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-3371316-0005.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-3371316-0005.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-6839027-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6839027-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-6839027-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-1335423-0010.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-1335423-0010.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-1335423-0010.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-6335562-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6335562-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-6335562-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-4815049-0003.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4815049-0003.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-4815049-0003.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-4320648-0003.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4320648-0003.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-4320648-0003.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-7432256-0003.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-7432256-0003.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-7432256-0003.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-463487-0007.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-463487-0007.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-463487-0007.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-4401421-0002.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-4401421-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-4401421-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-6823523-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6823523-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-6823523-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-5701772-0003.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5701772-0003.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-5701772-0003.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-3722564-0004.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-3722564-0004.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-3722564-0004.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-7299583-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-7299583-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-7299583-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-2814755-0004.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-2814755-0004.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-2814755-0004.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-463487-0004.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-463487-0004.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-463487-0004.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-7564600-0007.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-7564600-0007.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-7564600-0007.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-8241352-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8241352-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-8241352-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-8797622-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8797622-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-8797622-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-1546263-0004.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-1546263-0004.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-1546263-0004.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-2180985-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-2180985-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-2180985-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-5615173-0004.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5615173-0004.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-5615173-0004.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-3000214-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-3000214-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-3000214-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-741892-0003.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-741892-0003.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-741892-0003.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-1801584-0004.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-1801584-0004.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-1801584-0004.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-4016460-0004.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4016460-0004.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-4016460-0004.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-5221323-0004.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5221323-0004.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-5221323-0004.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-6669017-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6669017-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-6669017-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-2092977-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-2092977-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-2092977-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-5285130-0006.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5285130-0006.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-5285130-0006.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-7792467-0002.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-7792467-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-7792467-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-4671230-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4671230-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-4671230-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-1245228-0002.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-1245228-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-1245228-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-966858-0002.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-966858-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-966858-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-428676-0002.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-428676-0002.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/VIRUS-428676-0002.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-9332973-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-9332973-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-9332973-0001.jpeg
----------------------------
Moving sagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-649767-0001.jpeg to chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-649767-0001.jpeg
Deletingsagemaker-us-east-1-286375333242/chestCRXImages/chest_xray/train/PNEUMONIA/BACTERIA-649767-0001.jpeg

Now the data is ready on S3 and is plit into train, calibration, and test and ready for training.

Data Exploration

To keep the notebook simplier and easier to follow the data exploration is skipped. X-ray images would be shown later in the notebook when inference is done.

Early on exploration of the X-ray images showed that two main pre-processing need to happen:

  • Resizing the images because the size of input images can vary
  • Since there are BW and our pre-trained model is for 3-color channel images, we need to convert the images from BW to RGB. When using a dataloader from pytorch during the script training, this is done automatically - But when we want to send the image data to the endpoint for inference we need to explicity to this conversion as we will see later in the notebook.

Data Exploration

First, we do a hyper-parameter tuning of the model.

The training script is in CNN_training.py It is based on ResNet50 pre-trained model. The last dense/Fully-connected layer is replaced by two layers. One with 128 nodes and the last layer is with two nodes for the binary classification. The code is uploaded to the github folder.

For hyper-parameter tuning we can search for optimum values of learning rate and batch size, while aiming to minimize the testing loss.

Hyper-parameter Tuning

In [13]:
hyperparameter_ranges = {
    "learning_rate": ContinuousParameter(0.001, 0.01),
    "batch_size": CategoricalParameter([16, 32, 64, 128]),
}

role = sagemaker.get_execution_role()

objective_metric_name = "Test Loss"
objective_type = "Minimize"
metric_definitions = [{"Name": "Test Loss", "Regex": "Testing Loss: ([0-9\\.]+)"}]
In [14]:
estimator = PyTorch(
    entry_point="CNN_training.py",
    base_job_name='CRX_Pneumonia',
    role=role,
    framework_version="1.8",
    instance_count=1,
    instance_type="ml.m5.large",
    py_version='py36'
)

tuner = HyperparameterTuner(
    estimator,
    objective_metric_name,
    hyperparameter_ranges,
    metric_definitions,
    max_jobs=2,
    max_parallel_jobs=1,  # you once have one ml.g4dn.xlarge instance available
    objective_type=objective_type
)

Fit the tuner

In [14]:
os.environ['SM_CHANNEL_TRAINING']=s3datapath
os.environ['SM_MODEL_DIR']=os.path.join(s3datapath, "model")#'s3://udacitysolution/model/'
os.environ['SM_OUTPUT_DATA_DIR']=os.path.join(s3datapath, "output")#'s3://udacitysolution/output/'
tuner.fit({"training": s3datapath})
No finished training job found associated with this estimator. Please make sure this estimator is only used for building workflow config
No finished training job found associated with this estimator. Please make sure this estimator is only used for building workflow config
...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................!
In [15]:
#or find under sagemaker hyper-parameter tuning jobs if kernel dies

#latest_tuning_job_name = tuner.latest_tuning_job.name
latest_tuning_job_name = "pytorch-training-220921-0404"

print(latest_tuning_job_name)
pytorch-training-220921-0404

Tuning results

In [16]:
exp = HyperparameterTuningJobAnalytics(
  hyperparameter_tuning_job_name=latest_tuning_job_name)

jobs = exp.dataframe()

jobs.sort_values('FinalObjectiveValue', ascending=0)
Out[16]:
batch_size learning_rate TrainingJobName TrainingJobStatus FinalObjectiveValue TrainingStartTime TrainingEndTime TrainingElapsedTimeSeconds
1 "32" 0.003989 pytorch-training-220921-0404-001-55d34e19 Completed 14.0 2022-09-21 04:06:47+00:00 2022-09-21 04:46:59+00:00 2412.0
0 "32" 0.006130 pytorch-training-220921-0404-002-551b9194 Completed 12.0 2022-09-21 04:54:43+00:00 2022-09-21 05:34:03+00:00 2360.0

Imp: If kernel dies, load from sagemaker

In [17]:
BetterTrainingJobName='pytorch-training-220921-0404-002-551b9194'
my_estimator = sagemaker.estimator.Estimator.attach(BetterTrainingJobName)
my_estimator.hyperparameters()
best_estimator=my_estimator
2022-09-21 05:34:06 Starting - Found matching resource for reuse
2022-09-21 05:34:06 Downloading - Downloading input data
2022-09-21 05:34:06 Training - Training image download completed. Training in progress.
2022-09-21 05:34:06 Uploading - Uploading generated training model
2022-09-21 05:34:06 Completed - Resource released due to keep alive period expiry

Model training with the best hyper-parameters

In [ ]:
best_estimator=tuner.best_estimator()
In [18]:
best_estimator.hyperparameters()
Out[18]:
{'_tuning_objective_metric': 'Test Loss',
 'batch_size': '"32"',
 'learning_rate': '0.0061302077895518114',
 'sagemaker_container_log_level': '20',
 'sagemaker_estimator_class_name': '"PyTorch"',
 'sagemaker_estimator_module': '"sagemaker.pytorch.estimator"',
 'sagemaker_job_name': '"CRX_Pneumonia-2022-09-21-04-04-41-276"',
 'sagemaker_program': '"CNN_training.py"',
 'sagemaker_region': '"us-east-1"',
 'sagemaker_submit_directory': '"s3://sagemaker-us-east-1-286375333242/CRX_Pneumonia-2022-09-21-04-04-41-276/source/sourcedir.tar.gz"'}
In [19]:
hyperparameters = {"batch_size": int(best_estimator.hyperparameters()['batch_size'].replace('"', '')), \
                   "lr": best_estimator.hyperparameters()['learning_rate'],
                   "epochs" : 5}
hyperparameters
Out[19]:
{'batch_size': 32, 'lr': '0.0061302077895518114', 'epochs': 5}

Setting up debugger and profiler for the training job

In [20]:
rules = [
    Rule.sagemaker(rule_configs.vanishing_gradient()),
    Rule.sagemaker(rule_configs.overfit()),
    Rule.sagemaker(rule_configs.overtraining()),
    Rule.sagemaker(rule_configs.poor_weight_initialization()),
    ProfilerRule.sagemaker(rule_configs.ProfilerReport()),
]
In [21]:
hook_config = DebuggerHookConfig(
    hook_parameters={
        "train.save_interval": "1",
        "eval.save_interval": "1"
    }
)

profiler_config = ProfilerConfig(
    system_monitor_interval_millis=500, framework_profile_params=FrameworkProfile(num_steps=1)
)

Creating an Estimator

In [22]:
estimator = PyTorch(
    entry_point='CNN_training.py',
    base_job_name='CXR-Pneumonia',
    role=role,
    instance_count=1,
    instance_type='ml.p3.2xlarge',
    framework_version='1.8',
    py_version='py36',
    hyperparameters=hyperparameters,
    ## Debugger and Profiler parameters
    rules = rules,
    debugger_hook_config=hook_config,
    profiler_config=profiler_config,
)

Training the model

In [23]:
estimator.fit({"training": s3datapath}, wait = True)
2022-09-28 07:17:57 Starting - Starting the training job...
2022-09-28 07:18:23 Starting - Preparing the instances for trainingVanishingGradient: InProgress
Overfit: InProgress
Overtraining: InProgress
PoorWeightInitialization: InProgress
ProfilerReport: InProgress
.........
2022-09-28 07:19:57 Downloading - Downloading input data.........
2022-09-28 07:21:18 Training - Downloading the training image.....................
2022-09-28 07:24:59 Training - Training image download completed. Training in progress..bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
2022-09-28 07:24:56,447 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training
2022-09-28 07:24:56,479 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.
2022-09-28 07:24:56,486 sagemaker_pytorch_container.training INFO     Invoking user training script.
2022-09-28 07:24:57,036 sagemaker-training-toolkit INFO     Invoking user script
Training Env:
{
    "additional_framework_parameters": {},
    "channel_input_dirs": {
        "training": "/opt/ml/input/data/training"
    },
    "current_host": "algo-1",
    "framework_module": "sagemaker_pytorch_container.training:main",
    "hosts": [
        "algo-1"
    ],
    "hyperparameters": {
        "batch_size": 32,
        "epochs": 5,
        "lr": "0.0061302077895518114"
    },
    "input_config_dir": "/opt/ml/input/config",
    "input_data_config": {
        "training": {
            "TrainingInputMode": "File",
            "S3DistributionType": "FullyReplicated",
            "RecordWrapperType": "None"
        }
    },
    "input_dir": "/opt/ml/input",
    "is_master": true,
    "job_name": "CXR-Pneumonia-2022-09-28-07-17-57-174",
    "log_level": 20,
    "master_hostname": "algo-1",
    "model_dir": "/opt/ml/model",
    "module_dir": "s3://sagemaker-us-east-1-286375333242/CXR-Pneumonia-2022-09-28-07-17-57-174/source/sourcedir.tar.gz",
    "module_name": "CNN_training",
    "network_interface_name": "eth0",
    "num_cpus": 8,
    "num_gpus": 1,
    "output_data_dir": "/opt/ml/output/data",
    "output_dir": "/opt/ml/output",
    "output_intermediate_dir": "/opt/ml/output/intermediate",
    "resource_config": {
        "current_host": "algo-1",
        "current_instance_type": "ml.p3.2xlarge",
        "current_group_name": "homogeneousCluster",
        "hosts": [
            "algo-1"
        ],
        "instance_groups": [
            {
                "instance_group_name": "homogeneousCluster",
                "instance_type": "ml.p3.2xlarge",
                "hosts": [
                    "algo-1"
                ]
            }
        ],
        "network_interface_name": "eth0"
    },
    "user_entry_point": "CNN_training.py"
}
Environment variables:
SM_HOSTS=["algo-1"]
SM_NETWORK_INTERFACE_NAME=eth0
SM_HPS={"batch_size":32,"epochs":5,"lr":"0.0061302077895518114"}
SM_USER_ENTRY_POINT=CNN_training.py
SM_FRAMEWORK_PARAMS={}
SM_RESOURCE_CONFIG={"current_group_name":"homogeneousCluster","current_host":"algo-1","current_instance_type":"ml.p3.2xlarge","hosts":["algo-1"],"instance_groups":[{"hosts":["algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.p3.2xlarge"}],"network_interface_name":"eth0"}
SM_INPUT_DATA_CONFIG={"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}}
SM_OUTPUT_DATA_DIR=/opt/ml/output/data
SM_CHANNELS=["training"]
SM_CURRENT_HOST=algo-1
SM_MODULE_NAME=CNN_training
SM_LOG_LEVEL=20
SM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main
SM_INPUT_DIR=/opt/ml/input
SM_INPUT_CONFIG_DIR=/opt/ml/input/config
SM_OUTPUT_DIR=/opt/ml/output
SM_NUM_CPUS=8
SM_NUM_GPUS=1
SM_MODEL_DIR=/opt/ml/model
SM_MODULE_DIR=s3://sagemaker-us-east-1-286375333242/CXR-Pneumonia-2022-09-28-07-17-57-174/source/sourcedir.tar.gz
SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_pytorch_container.training:main","hosts":["algo-1"],"hyperparameters":{"batch_size":32,"epochs":5,"lr":"0.0061302077895518114"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"CXR-Pneumonia-2022-09-28-07-17-57-174","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://sagemaker-us-east-1-286375333242/CXR-Pneumonia-2022-09-28-07-17-57-174/source/sourcedir.tar.gz","module_name":"CNN_training","network_interface_name":"eth0","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_group_name":"homogeneousCluster","current_host":"algo-1","current_instance_type":"ml.p3.2xlarge","hosts":["algo-1"],"instance_groups":[{"hosts":["algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.p3.2xlarge"}],"network_interface_name":"eth0"},"user_entry_point":"CNN_training.py"}
SM_USER_ARGS=["--batch_size","32","--epochs","5","--lr","0.0061302077895518114"]
SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
SM_CHANNEL_TRAINING=/opt/ml/input/data/training
SM_HP_BATCH_SIZE=32
SM_HP_EPOCHS=5
SM_HP_LR=0.0061302077895518114
PYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python36.zip:/opt/conda/lib/python3.6:/opt/conda/lib/python3.6/lib-dynload:/opt/conda/lib/python3.6/site-packages
Invoking script with the following command:
/opt/conda/bin/python3.6 CNN_training.py --batch_size 32 --epochs 5 --lr 0.0061302077895518114
[2022-09-28 07:25:00.116 algo-1:26 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None
[2022-09-28 07:25:00.235 algo-1:26 INFO profiler_config_parser.py:102] Using config at /opt/ml/input/config/profilerconfig.json.
Namespace(batch_size=32, data='/opt/ml/input/data/training', epochs=5, lr=0.0061302077895518114, model_dir='/opt/ml/model', output_dir='/opt/ml/output/data', test_dir='/opt/ml/input/data/training/test', train_dir='/opt/ml/input/data/training/train')
Hyperparameters are LR: 0.0061302077895518114, Batch Size: 32
Epochs: 5
Data Paths: /opt/ml/input/data/training
Available device is: cuda:0
[2022-09-28 07:25:07.642 algo-1:26 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.
[2022-09-28 07:25:07.644 algo-1:26 INFO hook.py:201] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.
[2022-09-28 07:25:07.646 algo-1:26 INFO hook.py:255] Saving to /opt/ml/output/tensors
[2022-09-28 07:25:07.646 algo-1:26 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.
[2022-09-28 07:25:07.677 algo-1:26 INFO hook.py:591] name:fc.0.weight count_params:262144
[2022-09-28 07:25:07.678 algo-1:26 INFO hook.py:591] name:fc.0.bias count_params:128
[2022-09-28 07:25:07.678 algo-1:26 INFO hook.py:591] name:fc.2.weight count_params:256
[2022-09-28 07:25:07.679 algo-1:26 INFO hook.py:591] name:fc.2.bias count_params:2
[2022-09-28 07:25:07.679 algo-1:26 INFO hook.py:593] Total Trainable Params: 262530
Starting Model Training
Hyperparameters: epoch: 5, lr: 0.0061302077895518114, batch size: 32
[2022-09-28 07:25:08.710 algo-1:26 INFO hook.py:425] Monitoring the collections: gradients, relu_input, losses
[2022-09-28 07:25:08.712 algo-1:26 INFO python_profiler.py:182] Dumping cProfile stats to /opt/ml/output/profiler/framework/pytorch/cprofile/26-algo-1/prestepzero-*-start-1664349900235821.5_train-0-stepstart-1664349908711716.2/python_stats.
[2022-09-28 07:25:08.727 algo-1:26 INFO hook.py:488] Hook is writing from the hook with pid: 26
[2022-09-28 07:25:21.492 algo-1:26 INFO python_profiler.py:182] Dumping cProfile stats to /opt/ml/output/profiler/framework/pytorch/cprofile/26-algo-1/train-0-stepstart-1664349908723235.8_train-0-forwardpassend-1664349921492455.8/python_stats.
[2022-09-28 07:25:22.161 algo-1:26 INFO python_profiler.py:182] Dumping cProfile stats to /opt/ml/output/profiler/framework/pytorch/cprofile/26-algo-1/train-0-forwardpassend-1664349921495912.8_train-1-stepstart-1664349922160183.8/python_stats.
Train epoch 1: [320/5232 (6%)]#011Loss: 0.375405
Train epoch 1: [640/5232 (12%)]#011Loss: 0.206932
Train epoch 1: [960/5232 (18%)]#011Loss: 0.438136
Train epoch 1: [1280/5232 (24%)]#011Loss: 0.439663
Train epoch 1: [1600/5232 (30%)]#011Loss: 0.494468
Train epoch 1: [1920/5232 (37%)]#011Loss: 0.133440
Train epoch 1: [2240/5232 (43%)]#011Loss: 0.387662
Train epoch 1: [2560/5232 (49%)]#011Loss: 0.207986
Train epoch 1: [2880/5232 (55%)]#011Loss: 0.222797
Train epoch 1: [3200/5232 (61%)]#011Loss: 0.223536
Train epoch 1: [3520/5232 (67%)]#011Loss: 0.493388
Train epoch 1: [3840/5232 (73%)]#011Loss: 0.152841
Train epoch 1: [4160/5232 (79%)]#011Loss: 0.514780
Train epoch 1: [4480/5232 (85%)]#011Loss: 0.292197
Train epoch 1: [4800/5232 (91%)]#011Loss: 0.285628
Train epoch 1: [5120/5232 (98%)]#011Loss: 0.224347
Testing...
Test set: Average loss: 0.0134, Test Accuracy: 84.61538461538461%
Train epoch 2: [320/5232 (6%)]#011Loss: 0.143077
Train epoch 2: [640/5232 (12%)]#011Loss: 0.202894
Train epoch 2: [960/5232 (18%)]#011Loss: 0.096761
Train epoch 2: [1280/5232 (24%)]#011Loss: 0.292739
Train epoch 2: [1600/5232 (30%)]#011Loss: 0.098056
Train epoch 2: [1920/5232 (37%)]#011Loss: 0.131096
Train epoch 2: [2240/5232 (43%)]#011Loss: 0.163605
Train epoch 2: [2560/5232 (49%)]#011Loss: 0.290103
Train epoch 2: [2880/5232 (55%)]#011Loss: 0.465909
Train epoch 2: [3200/5232 (61%)]#011Loss: 0.327568
Train epoch 2: [3520/5232 (67%)]#011Loss: 0.215384
Train epoch 2: [3840/5232 (73%)]#011Loss: 0.339750
Train epoch 2: [4160/5232 (79%)]#011Loss: 0.301169
Train epoch 2: [4480/5232 (85%)]#011Loss: 0.177459
Train epoch 2: [4800/5232 (91%)]#011Loss: 0.140137
Train epoch 2: [5120/5232 (98%)]#011Loss: 0.334776
Testing...
Test set: Average loss: 0.0105, Test Accuracy: 85.25641025641026%
Train epoch 3: [320/5232 (6%)]#011Loss: 0.320791
Train epoch 3: [640/5232 (12%)]#011Loss: 0.113965
Train epoch 3: [960/5232 (18%)]#011Loss: 0.328466
Train epoch 3: [1280/5232 (24%)]#011Loss: 0.139991
Train epoch 3: [1600/5232 (30%)]#011Loss: 0.213227
Train epoch 3: [1920/5232 (37%)]#011Loss: 0.265513
Train epoch 3: [2240/5232 (43%)]#011Loss: 0.318187
Train epoch 3: [2560/5232 (49%)]#011Loss: 0.189421
Train epoch 3: [2880/5232 (55%)]#011Loss: 0.247238
Train epoch 3: [3200/5232 (61%)]#011Loss: 0.358383
Train epoch 3: [3520/5232 (67%)]#011Loss: 0.305215
Train epoch 3: [3840/5232 (73%)]#011Loss: 0.225445
Train epoch 3: [4160/5232 (79%)]#011Loss: 0.175306
Train epoch 3: [4480/5232 (85%)]#011Loss: 0.355496
Train epoch 3: [4800/5232 (91%)]#011Loss: 0.203509
Train epoch 3: [5120/5232 (98%)]#011Loss: 0.329521
Testing...
Test set: Average loss: 0.0107, Test Accuracy: 84.61538461538461%
Train epoch 4: [320/5232 (6%)]#011Loss: 0.404709
VanishingGradient: InProgress
Overfit: IssuesFound
Overtraining: InProgress
PoorWeightInitialization: InProgress
Train epoch 4: [640/5232 (12%)]#011Loss: 0.227930
Train epoch 4: [960/5232 (18%)]#011Loss: 0.142792
Train epoch 4: [1280/5232 (24%)]#011Loss: 0.253626
Train epoch 4: [1600/5232 (30%)]#011Loss: 0.301804
Train epoch 4: [1920/5232 (37%)]#011Loss: 0.212710
Train epoch 4: [2240/5232 (43%)]#011Loss: 0.214941
Train epoch 4: [2560/5232 (49%)]#011Loss: 0.316854
Train epoch 4: [2880/5232 (55%)]#011Loss: 0.128724
Train epoch 4: [3200/5232 (61%)]#011Loss: 0.253114
Train epoch 4: [3520/5232 (67%)]#011Loss: 0.312859
Train epoch 4: [3840/5232 (73%)]#011Loss: 0.294126
Train epoch 4: [4160/5232 (79%)]#011Loss: 0.204234
Train epoch 4: [4480/5232 (85%)]#011Loss: 0.131046
Train epoch 4: [4800/5232 (91%)]#011Loss: 0.227132
Train epoch 4: [5120/5232 (98%)]#011Loss: 0.141236
Testing...
VanishingGradient: InProgress
Overfit: IssuesFound
Overtraining: IssuesFound
PoorWeightInitialization: InProgress
Test set: Average loss: 0.0193, Test Accuracy: 77.40384615384616%
Train epoch 5: [320/5232 (6%)]#011Loss: 0.346485
Train epoch 5: [640/5232 (12%)]#011Loss: 0.147757
Train epoch 5: [960/5232 (18%)]#011Loss: 0.200689
Train epoch 5: [1280/5232 (24%)]#011Loss: 0.216724
Train epoch 5: [1600/5232 (30%)]#011Loss: 0.276936
Train epoch 5: [1920/5232 (37%)]#011Loss: 0.228897
Train epoch 5: [2240/5232 (43%)]#011Loss: 0.488071
Train epoch 5: [2560/5232 (49%)]#011Loss: 0.228150
Train epoch 5: [2880/5232 (55%)]#011Loss: 0.442228
Train epoch 5: [3200/5232 (61%)]#011Loss: 0.426490
Train epoch 5: [3520/5232 (67%)]#011Loss: 0.103515
Train epoch 5: [3840/5232 (73%)]#011Loss: 0.224388
Train epoch 5: [4160/5232 (79%)]#011Loss: 0.201727
Train epoch 5: [4480/5232 (85%)]#011Loss: 0.154246
Train epoch 5: [4800/5232 (91%)]#011Loss: 0.397623
Train epoch 5: [5120/5232 (98%)]#011Loss: 0.293890
Testing...

2022-09-28 07:35:16 Uploading - Uploading generated training modelTest set: Average loss: 0.0131, Test Accuracy: 85.41666666666667%
Saving the model to /opt/ml/model/model.pth
Training Finished...
INFO:__main__:Hyperparameters are LR: 0.0061302077895518114, Batch Size: 32
INFO:__main__:Epochs: 5
INFO:__main__:Data Paths: /opt/ml/input/data/training
Downloading: "https://download.pytorch.org/models/resnet50-19c8e357.pth" to /root/.cache/torch/hub/checkpoints/resnet50-19c8e357.pth
#015  0%|          | 0.00/97.8M [00:00<?, ?B/s]#015  5%|▌         | 5.18M/97.8M [00:00<00:01, 54.3MB/s]#015 13%|█▎        | 12.2M/97.8M [00:00<00:01, 66.0MB/s]#015 20%|█▉        | 19.3M/97.8M [00:00<00:01, 69.7MB/s]#015 27%|██▋       | 26.4M/97.8M [00:00<00:01, 71.3MB/s]#015 34%|███▍      | 33.2M/97.8M [00:00<00:00, 71.5MB/s]#015 41%|████▏     | 40.4M/97.8M [00:00<00:00, 72.6MB/s]#015 48%|████▊     | 47.4M/97.8M [00:00<00:00, 72.9MB/s]#015 56%|█████▌    | 54.5M/97.8M [00:00<00:00, 73.3MB/s]#015 63%|██████▎   | 61.5M/97.8M [00:00<00:00, 72.5MB/s]#015 70%|███████   | 68.6M/97.8M [00:01<00:00, 73.1MB/s]#015 77%|███████▋  | 75.7M/97.8M [00:01<00:00, 73.5MB/s]#015 85%|████████▍ | 82.9M/97.8M [00:01<00:00, 74.1MB/s]#015 92%|█████████▏| 90.0M/97.8M [00:01<00:00, 74.4MB/s]#015 99%|█████████▉| 97.1M/97.8M [00:01<00:00, 73.9MB/s]#015100%|██████████| 97.8M/97.8M [00:01<00:00, 72.3MB/s]
INFO:__main__:Starting Model Training
INFO:__main__:Testing...
INFO:__main__:
Test set: Average loss: 0.0134, Test Accuracy: 84.61538461538461%
INFO:__main__:Testing...
INFO:__main__:
Test set: Average loss: 0.0105, Test Accuracy: 85.25641025641026%
INFO:__main__:Testing...
INFO:__main__:
Test set: Average loss: 0.0107, Test Accuracy: 84.61538461538461%
INFO:__main__:Testing...
INFO:__main__:
Test set: Average loss: 0.0193, Test Accuracy: 77.40384615384616%
INFO:__main__:Testing...
INFO:__main__:
Test set: Average loss: 0.0131, Test Accuracy: 85.41666666666667%
INFO:__main__:Saving the model to /opt/ml/model/model.pth
2022-09-28 07:35:06,554 sagemaker-training-toolkit INFO     Reporting training SUCCESS

2022-09-28 07:35:42 Completed - Training job completed
ProfilerReport: IssuesFound
Training seconds: 955
Billable seconds: 955

Training results

We get around 85% accuracy compared to 90% from the benchmark paper. We could perhaps make the model deeper or increase the training loss to improve the result. It could also be that since we dedicated almost 10% of our Pneumonia images for probability calibration, we did not have enough positive samples for the model to learn.

However, we could also be satisfied with the result. Although, the benchmark reports higher accuracy it does not necessarily be a good model for predicting the probability of Pneumonia in a patient.

Deploy the model

We deploy the model to an endpoint to use it for inference and calibration:

In [24]:
model_location=estimator.model_data
print("Mode data in case of reload or if the kernel dies before deployment:")
print(model_location)
Mode data in case of reload or if the kernel dies before deployment:
s3://sagemaker-us-east-1-286375333242/CXR-Pneumonia-2022-09-28-07-17-57-174/output/model.tar.gz
In [17]:
import sagemaker
import boto3
from sagemaker.tuner import CategoricalParameter, ContinuousParameter, HyperparameterTuner
from sagemaker.pytorch import PyTorch
from sagemaker import get_execution_role
from sagemaker.debugger import Rule, DebuggerHookConfig, TensorBoardOutputConfig, CollectionConfig, ProfilerRule, rule_configs
from sagemaker.debugger import ProfilerConfig, FrameworkProfile

from sagemaker.pytorch import PyTorchModel
from sagemaker.predictor import Predictor
In [18]:
jpeg_serializer = sagemaker.serializers.IdentitySerializer("image/jpeg")
json_deserializer = sagemaker.deserializers.JSONDeserializer()

class ImagePredictor(Predictor):
    def __init__(self, endpoint_name, sagemaker_session):
        super(ImagePredictor, self).__init__(
            endpoint_name,
            sagemaker_session=sagemaker_session,
            serializer=jpeg_serializer,
            deserializer=json_deserializer,
        )
In [27]:
pytorch_model = PyTorchModel(model_data=model_location, role=role, entry_point='inference.py',py_version='py36',
                             framework_version='1.8',
                             predictor_cls=ImagePredictor)
In [28]:
predictor = pytorch_model.deploy(initial_instance_count=1, instance_type='ml.m5.large')
------!
In [14]:
#endpoint_name = predictor.endpoint_name
endpoint_name = "pytorch-inference-2022-09-28-07-40-07-421"

print("The endpoint name is:")
print(endpoint_name)
The endpoint name is:
pytorch-inference-2022-09-28-07-40-07-421

Run this cell if the kernel dies but the endpoint is still active:

In [19]:
predictor = sagemaker.predictor.Predictor(
                                          endpoint_name = endpoint_name,
                                          sagemaker_session = session,
                                          serializer = jpeg_serializer,
                                          deserializer = json_deserializer  
                                          )
In [ ]:
 

Probability Calibration

Now, we start on training the calibration model. First, we create a list of all the X-ray images we have available for training the model:

In [9]:
s3 = boto3.resource('s3')
my_bucket = s3.Bucket(bucket)
Normal_images_calibration = []

for object_summary in my_bucket.objects.filter(Prefix="chestCRXImages/chest_xray/Calibration/NORMAL"):
    #print(object_summary.key)
    Normal_images_calibration.append(object_summary.key)
    

Pneumonia_images_calibration = []
for object_summary in my_bucket.objects.filter(Prefix="chestCRXImages/chest_xray/Calibration/PNEUMONIA"):
    #print(object_summary.key)
    Pneumonia_images_calibration.append(object_summary.key)
    
print(f"Number of Normal Images available for calibration: {len(Normal_images_calibration)}")
print(f"Number of PNEUMONIA Images available for calibration: {len(Pneumonia_images_calibration)}")
Number of Normal Images available for calibration: 99
Number of PNEUMONIA Images available for calibration: 99

Creating a list of all the images (to be sent to the endpoint for inference) and their labels (from their folder)

In [10]:
cal_data = []
cal_y = []
for image_name in Normal_images_calibration:
    image_label = 0
    cal_data.append(image_name)
    cal_y.append(image_label)
    #print('----------------------------')
    #print("Image: " + image_name)
    #print("Label: " + str(image_label))

for image_name in Pneumonia_images_calibration:
    image_label = 1
    cal_data.append(image_name)
    cal_y.append(image_label)
    #print('----------------------------')
    #print("Image: " + image_name)
    #print("Label: " + str(image_label))

Testing the endpoint and visualizing the results

First step is to send the images in the calibration dataset to the model (endpoint) and calulate the probability of the Pneumonia.

Since the last of the CNN was outputing the logits, we need to take the softmax to get the probabilty.

We store the uncalibrated probability of the entire calibration dataset into a list.

For visualization for the first 3 images the X-ray image is shown. For the rest only the predicted label (NORMAL vs. PNEUMONIA) and the uncalibrated probability of PNEUMONIA is printed.

In [12]:
def softmax(x):
    return np.exp(x) / np.sum(np.exp(x), axis=0)
In [20]:
from sagemaker.serializers import IdentitySerializer
from PIL import Image

predictor.serializer = IdentitySerializer("image/jpeg")
jpeg_serializer = IdentitySerializer("image/jpeg")

session = boto3.Session()
s3_client = session.client('s3')
s3 = boto3.resource('s3')   

bucket_name = bucket
bucket_ = s3.Bucket(bucket_name)
count = 0
uncalibrated_Pneumonia_probabilities_cal = []
for file, label in zip(cal_data, cal_y):
    print("--------------------------")
    print("Image: " + file)
    print("Label: " + str(label))
    object = bucket_.Object(file)
    response = object.get()
    file_stream = response['Body']
    img = Image.open(file_stream).convert('RGB')
    if count < 3:
        display(img)
    count += 1
    
    s3_object = s3_client.get_object(Bucket = bucket_name, Key = file)
    body = s3_object['Body']
    content = body.read()
    response = predictor.predict(content, initial_args={'ContentType':'image/jpeg'})
    probs = softmax(response[0])
    uncalibrated_Pneumonia_probability = probs[1]
    print("Uncalibrated Pneumonia Probability: " + str(uncalibrated_Pneumonia_probability))
    uncalibrated_Pneumonia_probabilities_cal.append(uncalibrated_Pneumonia_probability)
    prediction = np.argmax(response[0]) + 1 
    prediction = 'NORMAL' if np.argmax(response[0]) == 0 else 'PNEUMONIA' 
    print("Prediction is: " + prediction)
    
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1044645-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.08247949068642986
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1101909-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5358327650468722
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1128157-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5488907091728171
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1455093-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1992966265915107
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1522956-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1827606288369004
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1602738-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.06274437966081287
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1696866-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.055224822244688776
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1739213-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.15349568412461392
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1756042-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.044498276455786676
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1830070-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.40198248212398513
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-1917089-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.14148137452643758
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-202916-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.09611644921460302
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-2048417-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.17898001218638768
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-2239132-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.39087523194416035
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-2239455-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.04567903905561678
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-2285600-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.18722167000483841
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-229022-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.16722136230845724
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-2333546-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.24213955750321503
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-2387301-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.11582766561999111
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-263932-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.11883243416140077
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3072074-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.21413455127489442
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3175613-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9768667963699771
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-32326-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.40992600338892177
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3487615-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.18149718792345834
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3573766-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.07058099275082463
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3644944-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2045191648019463
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3747940-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.17270908559364215
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3781678-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.04859008207883672
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3789615-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.07524698076502394
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3821719-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.11818122698375742
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3852346-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3163569940923866
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3964800-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4257504588453206
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-3997421-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.12050903460892232
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4190635-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2769702483758563
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4221981-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.03318439940403991
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4271375-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1487281914933459
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4507677-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10719531134798634
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-452722-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.16879908978989075
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4758820-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1391297201624191
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4791400-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.44083880262965225
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-4969331-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.08543475931914198
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5031579-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.27764134768713755
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5042453-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.15900857036103158
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5176049-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.6265080514765197
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5236765-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.19394877993559873
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5243995-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.34279450465192246
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-546828-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1351660948880327
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5602301-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9266349934749432
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5650764-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.16890664290005175
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5708209-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.6191574639786482
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5728847-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10184637376689493
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5759621-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.14386811741520833
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-591136-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.05443430324445407
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-5918516-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.09368743095945177
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6044251-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.14998640544542843
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6081265-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10345038597057617
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6112892-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.33716113240070167
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6459868-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.28618963889596305
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6830488-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.023836801290251033
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6901859-0004.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.14631140026944728
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6915237-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.15009253393944647
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6973172-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2179695000024495
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-6978858-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.31012884551926634
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7001302-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.07397612136526702
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7019096-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.11799577966372278
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7103127-0003.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1040081669573553
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-715046-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.16665594220857846
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7298141-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1507403553019537
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7436212-0003.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.08336579352185146
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7451061-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.45792367243938403
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7575347-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.17249847573438598
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-7696091-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.056311185851588674
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8033319-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.0487115187933637
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8116905-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10552240784133801
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8143931-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.055859041368373365
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8181293-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.0687771835432738
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8229007-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.14495614973038862
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-824917-0003.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.14831707139588418
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8272933-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9809221099662029
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8412621-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1708985161044502
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8439083-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.04494916466057478
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8644314-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.22906245390500732
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8654141-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.030356048050112227
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8688367-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10181833078155411
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8799208-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.7869254936532897
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8825351-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.09508054589455592
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8903542-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.26030302705502567
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-8998369-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.13035805894171412
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9033279-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5830863156152283
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9084382-0003.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1133765398920874
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-933709-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.18923549211005983
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9430515-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.05797977427030519
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9571602-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.15681203457769088
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9585882-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.20611053541580168
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9697349-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.37376775775751203
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9742274-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.396216510792556
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-974509-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.06368984913850202
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9814816-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1561747034367301
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/NORMAL/NORMAL-9816293-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.057285803587118614
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-1086099-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9981553935078319
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-1150345-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997655531671225
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-1170996-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.984739469872657
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-1546263-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5336009581878914
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-1612812-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9987920150881637
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-1620679-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9965266037370154
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-1865585-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9992025606191263
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-2092977-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8647898048541489
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-2341120-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.989019291461251
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-2355867-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9989332187267973
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-292199-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9998715264589907
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-3000214-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996011094816356
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-3067831-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997341959390899
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-3371316-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993022080147703
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-387616-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.998878604566256
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4016460-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5867859651337504
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4241544-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8732625157959566
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4320648-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.99296979261173
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4519487-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9924192044439597
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4615614-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9881791578825608
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4671230-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9973669470522887
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4766617-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.982326922839557
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-4815049-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994677899826245
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5121478-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9914795677882827
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5150733-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9885862178307646
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-516152-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9094030080212272
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5164118-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9818257699276405
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5285130-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9837420264001984
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5401887-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9691569749885933
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5437440-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9992350427264787
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5437440-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999813929883065
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5465856-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999270502942858
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5503803-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.625701318723582
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5678659-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9127759742159963
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5724853-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9862907958626117
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-5839760-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.3198851468210602
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6065052-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9948127926421738
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6335562-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999570180784473
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-649767-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9407494087426158
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6512753-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9728592531821827
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6592427-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5997886365743098
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6610440-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.767732599785662
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6669017-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997597419916731
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6823523-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999591161250703
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-6839027-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9846301273913991
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-7035571-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9959121729271968
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-741892-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5007508354205628
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-7432256-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9958268396857259
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-7676769-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9860810252793689
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-7721065-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9365190127905303
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-7746316-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9992057387873071
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-7882471-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9506276753112615
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8034950-0012.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9092655942122009
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8241352-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996133598925245
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8313249-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9957078026325431
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8498464-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999761711292495
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8545367-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996425048994109
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8660561-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9985293157369308
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-8797622-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9922640097350709
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-9026424-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9991413401990586
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-9269035-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5087190790672349
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-9332973-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9899304322032237
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-9487238-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9913062823629186
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-9566232-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9965060815891819
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-960851-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9963740128096463
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/BACTERIA-9674936-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994489579914886
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-1191925-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9375112197644844
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-1245228-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.911324861881724
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-1335423-0010.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9985639623127337
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-1801584-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9352787429084739
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-2110654-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8601197608104668
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-2180985-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.3780564916764594
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-2814755-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9904472507750794
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-324273-0008.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9886667599131377
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-3722564-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999492805626032
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-3829565-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993382617234827
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-4057077-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9408004461690002
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-4271267-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9035972745746248
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-428676-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.993536614259515
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-4401421-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9924479588208645
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-4617189-0009.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9907120525642635
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-463487-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9931707488230604
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-463487-0007.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9981845492718253
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5066255-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9981793871548416
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5150733-0015.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8872403178817235
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5221323-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9953766784821969
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5285130-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993319796974999
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5331068-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9955521186663538
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5615173-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993523885328185
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5701772-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9988943978172292
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-5882850-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8707910850304751
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-6718152-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9388836772150617
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-7299583-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993855628937748
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-7564600-0007.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.6386828012668863
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-7792467-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9985502199027168
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-7986969-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9812640620318392
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-8268782-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9893356151283801
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-8797622-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.4864939085443749
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/Calibration/PNEUMONIA/VIRUS-966858-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9742854952603427
Prediction is: PNEUMONIA

The calibration dataset does not have many samples and its visualization may not show enough information. However, we can still get an idea before looking into the entire test dataset and before calibration the probabilities.

We can see the most of the images result in either high probability or low probability, however these regions are important for calibration purposes and cannot be ignored as most of the physician errors coudl come from these areas. For other sections with not much points the data could be noisy.

In general from the figure we can learn that our current model over-estimated the probability of Pneumonia from the X-ray images as the predicted probabilties are generally higher than actual probabilties.

In [22]:
import matplotlib.pyplot as plt
from sklearn.calibration import calibration_curve, CalibrationDisplay


plt.figure(figsize=(10,5))
n, bins, patches = plt.hist(x=uncalibrated_Pneumonia_probabilities_cal, bins=5, color='#0504aa', alpha=0.5, rwidth=0.75)

plt.figure(figsize=(10,5))
prob_true, prob_pred = calibration_curve(cal_y, uncalibrated_Pneumonia_probabilities_cal, n_bins=5)

disp = CalibrationDisplay(prob_true, prob_pred, uncalibrated_Pneumonia_probabilities_cal)
disp.plot()
Out[22]:
<sklearn.calibration.CalibrationDisplay at 0x7f55340ffca0>
<Figure size 720x360 with 0 Axes>

Beta Calibration:

We send the uncalibrated Pneumonia probabilties and the true label for each image to a Beta Calibration model that we imported to train the calibration model.

There are several other calibration models such as Platt, Spline, and Isotonic.

For the purpose of this project we only focus on beta calibration.

After fitting the model, we can then use the predict method of the model and use it to convert the uncalibrated probability of the CNN into a calibrated Probability.

Note:

We could do all these inside the training script and deploy the calibration model as part of the endpoint. The calibration dataset could be sent into the training instance in a new channel and everything would work the same way. However, we could not have the visualization and follow the steps. In real production, we would do that as part of the script.

In [40]:
from betacal import BetaCalibration

betaCalibration = BetaCalibration()
betaCalibration.fit(uncalibrated_Pneumonia_probabilities_cal, cal_y)
Out[40]:
BetaCalibration()

Visualizing the results

To best compare the results we plot probability diagram for uncalibrated and calibrated probabilties of Pneumonia on the entire test dataset. For this we need to call the endpoint on each image on the test dataset.

Creating the list of images in the test dataset together with their labels:

In [24]:
s3 = boto3.resource('s3')
my_bucket = s3.Bucket(bucket)
Normal_images_test = []

for object_summary in my_bucket.objects.filter(Prefix="chestCRXImages/chest_xray/test/NORMAL"):
    Normal_images_test.append(object_summary.key)

Pneumonia_images_test = []
for object_summary in my_bucket.objects.filter(Prefix="chestCRXImages/chest_xray/test/PNEUMONIA"):
    Pneumonia_images_test.append(object_summary.key)
    
print(f"Number of Normal Images available for test: {len(Normal_images_test)}")
print(f"Number of PNEUMONIA Images available for test: {len(Pneumonia_images_test)}")
Number of Normal Images available for test: 234
Number of PNEUMONIA Images available for test: 390
In [27]:
test_data = []
test_y = []
for image_name in Normal_images_test:
    image_label = 0
    test_data.append(image_name)
    test_y.append(image_label)
    #print('----------------------------')
    #print("Image: " + image_name)
    #print("Label: " + str(image_label))

for image_name in Pneumonia_images_test:
    image_label = 1
    test_data.append(image_name)
    test_y.append(image_label)
    #print('----------------------------')
    #print("Image: " + image_name)
    #print("Label: " + str(image_label))

print(f"Total size of test dataset is {len(test_data)}")
Total size of test dataset is 624

Calling the endpoint and storing the calculated uncalibrated Pneumonia probabilties. We skip visualization of the images here and will do it at the end.

In [30]:
from sagemaker.serializers import IdentitySerializer

predictor.serializer = IdentitySerializer("image/jpeg")
jpeg_serializer = IdentitySerializer("image/jpeg")

session = boto3.Session()
s3_client = session.client('s3')
s3 = boto3.resource('s3')   

from PIL import Image
import numpy as np
bucket_name = bucket
bucket_ = s3.Bucket(bucket_name)
count = 0
uncalibrated_Pneumonia_probabilities_test = []
for file, label in zip(test_data, test_y):
    print("--------------------------")
    print("Image: " + file)
    print("Label: " + str(label))
    object = bucket_.Object(file)
    response = object.get()
    file_stream = response['Body']
    img = Image.open(file_stream).convert('RGB')
    #display(img)
    count += 1
    
    s3_object = s3_client.get_object(Bucket = bucket_name, Key = file)
    body = s3_object['Body']
    content = body.read()
    response = predictor.predict(content, initial_args={'ContentType':'image/jpeg'})
    probs = softmax(response[0])
    uncalibrated_Pneumonia_probability = probs[1]
    print("Uncalibrated Pneumonia Probability: " + str(uncalibrated_Pneumonia_probability))
    uncalibrated_Pneumonia_probabilities_test.append(uncalibrated_Pneumonia_probability)
    prediction = np.argmax(response[0]) + 1 
    prediction = 'NORMAL' if np.argmax(response[0]) == 0 else 'PNEUMONIA' 
    print("Prediction is: " + prediction)
    #if count == 5:
     #   break
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1049278-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8147431014939452
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1110860-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.26126209169732845
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-11419-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10093189776727404
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-115218-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4459524603035636
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1160949-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9375761720203815
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1212407-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2524494160190298
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1228182-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.21887118664468924
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1283091-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5309149848161783
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1318320-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10242404246439496
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1368583-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.6164154598953595
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1430636-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1519676408172442
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1520670-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.11014992222202792
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-152130-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.21373022107169332
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-159472-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9345202677279567
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1608079-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9762664163387705
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1627110-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.7268780915613468
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1698651-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.978289887387899
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-171327-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.18345303935393395
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1759114-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.36858404963792324
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1763721-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1443789139696456
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1768815-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.976976935532641
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1771524-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.35856522902480453
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1784004-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.46380132593226253
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1803887-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8315302686103259
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1858497-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.16476152207102582
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-186900-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.265372790761402
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1931427-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2930831821553782
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-1944537-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3002217690072368
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2107985-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.6332501028370436
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2123652-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9760612924888924
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2162145-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.19797389962793338
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-217318-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.37526594243407335
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2233350-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.13239548434412785
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2256620-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.6033092961914704
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2274324-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.12007627097223263
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2280080-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.042962616160498825
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2298727-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9857538652842233
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2403676-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9628586540416588
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2477476-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.0638413965578636
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2514572-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4078806510397887
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2578531-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.03170025980681353
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2704875-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.28001791562954287
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2728578-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3154354878562317
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2771774-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5180563678005933
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2798826-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.05201155358082771
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-283245-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.38854147302362735
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-284601-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9920142834911609
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2850034-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.08278823443680658
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2862975-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.6761836855274607
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2870844-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.20193618647430034
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2924895-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4135946463842736
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-2959018-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.17744641824295235
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3065672-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.39407156067816007
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3098476-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.17573135080152302
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3117193-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9440120494706454
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3170711-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.43976038648723165
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3189250-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9972815864641986
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3267425-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.365124189924618
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3293997-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.06868912045963804
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3322209-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1897355955822874
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3346259-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9642868165716043
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3395085-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.14283662891002313
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3396581-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.21157607009893642
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3397442-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.140156119121892
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3437568-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10321796701179742
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3459196-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.09328614996752617
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3464500-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2851896714312553
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3464500-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.22767441342768066
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3464500-0003.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3359567937172916
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3487620-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.38129234282062396
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3518268-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9245753435292091
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3549035-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9898475045372126
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3564811-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.6810926003605003
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3624496-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9957235806151649
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3686746-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.843330925753199
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3733696-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.07709330074722301
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3782395-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5142490268982204
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3837280-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.048200537789113866
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3921425-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.27429890843935034
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3937418-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.33227683928697466
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3983280-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.908456260955276
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-3985743-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2049376037784618
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4172448-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.22344221180303042
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4190414-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.16456558815374336
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4196521-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2892604179551749
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4315572-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8219498789855713
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4340661-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.13481373652815634
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4363211-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10433514657150718
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4511610-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.769134441148862
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4512-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3070449684560613
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4513268-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.26040760394821894
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4534031-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.27698429164167554
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4675782-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.6507691556177806
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-472699-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2735235301809359
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4766279-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.862800325821687
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4834475-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5477097262769715
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4852815-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.11713456827262474
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4886177-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4226393142728031
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4906464-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10573509360890299
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4911367-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.28669966723794466
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4921478-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.7678172197039417
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4955194-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10831415201435228
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5083836-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2553316352811159
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5148117-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2523385386994942
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5175014-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.11484184038421995
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5180957-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9127694442101764
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5200363-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.048289226528965
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-520267-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.25556613624595725
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-522391-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5461395451924254
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-528189-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.21020871529707272
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5349917-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4918206441663985
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5356114-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9471396834609814
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5388550-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.814555423391448
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5401638-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.11198163974982821
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5402107-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.07591389411361453
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-541071-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.08116303030870284
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5423277-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.07466280170041485
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5425295-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.983102686564178
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5459424-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.08715806241189292
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5472638-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3260230515834349
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5487025-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9931548643203468
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5498455-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1627692639811311
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5511958-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.453181737091916
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5513974-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1698690182355527
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5538803-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.15228517228310998
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5545906-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1851245285854423
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5578360-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5069226668992202
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5601980-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.6822533915251328
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5611875-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2049739868495768
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-561684-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.0945540713035013
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5661793-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.12227634052492262
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5661793-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1491160343484463
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5714306-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.09463388636228942
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5789014-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.15518888554489552
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5889760-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8342420243771473
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5893138-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1756405209223346
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-5997885-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.052791749458375094
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6000527-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.09699826717229462
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6085151-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3195373370092793
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6106856-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1243645872463348
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6112608-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.30122605256040264
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6137647-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9087077914189228
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6190523-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5040076112807167
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6195450-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2915127988449494
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6204576-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.10142495887490532
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6267994-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.44415413634625206
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6293151-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.25477114384502664
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6297597-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.22382167937194483
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-630880-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8712555385954821
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6322108-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.20029447046682788
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6332708-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.24607407536991396
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6332708-0002.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.822437452530241
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6332708-0003.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4324461681666481
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6342197-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8237882279833231
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6395296-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.7172762632109436
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6410283-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.11556811750971614
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6442221-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9295618513437252
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6497443-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.35632729502169824
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6503977-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.09121222038711452
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6518197-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9912148723805888
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6524124-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9877993935389832
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6544436-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.34710495290322385
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6569259-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.21005341097651145
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6650392-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9254488423286533
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6667079-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9100050037698492
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-667173-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9200931205548203
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6703189-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.30870137799061104
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6713743-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.53785497416208
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6874528-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2055223909065608
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6983998-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.15313650355676323
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7031638-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4681656819303341
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-706052-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.25836944356669617
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7082548-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8355077451188432
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-71002-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5977062219044457
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7100645-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8713761626415881
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7165123-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9507210344862049
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7225301-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8927948595203287
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7236364-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4111279961933971
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-724740-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2544369983918707
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7331861-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.22890392726274433
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7398200-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3667735081862521
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7423096-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9843174177807116
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-745902-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3163476046783968
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7483489-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.21924165995828726
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7488601-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.29772975239642463
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7519498-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.24345955120513557
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-754920-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4102456482602127
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7590045-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8714675639059734
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7695736-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.21611631450232602
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7738170-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.33602031735990134
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7804528-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.18346192353529947
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-78265-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.6264462588192173
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-786023-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9771310111321301
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7922918-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.18726456138406694
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7950798-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.044592857754042314
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7961936-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.056046146140161554
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8014191-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.08146008965691905
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8207090-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.19366237490024027
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8278998-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9918286315951118
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8305321-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.08095412725713258
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8305499-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4722195214414475
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8406720-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.956541072630444
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8417023-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9729955636696525
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8418332-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.07850138815548977
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8441235-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.0835237102298112
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8448068-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3812534607323749
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8495317-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9519470959070799
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-857568-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3756656779358182
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8586608-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2000505918428751
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8661138-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.171821953929319
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8698006-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4143422915852774
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-870639-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.19106033707315775
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8750803-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.7517363691228751
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8836138-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.7431183291168711
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8876914-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.1492213070095748
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8907762-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5122784272886866
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-8944072-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.20706083577045395
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9054075-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9988483665188695
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9077486-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.4866504365759639
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9092354-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.26409409889982344
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9104493-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.08976426674599758
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9145111-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.2770667994754604
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9194822-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.21487035554687783
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9217241-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3136003501731313
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9260113-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.06763408000562773
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9275512-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.5948746625600907
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9334007-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3127383690140007
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9343440-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.07740189207238869
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9441169-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.3337086115746784
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9708691-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.16600538727010342
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9717513-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.18193420497797
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-976006-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.7088676103370273
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9799657-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8186478499830165
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-9896495-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.07674111117809021
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1135262-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9724986771549766
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1135262-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9817690262076074
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1135262-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9968367721523199
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1135262-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9373627765046999
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1220485-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994570371002236
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1351146-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9992081429207745
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1351146-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999695349815599
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1351146-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9955128008658252
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1351146-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999237508582602
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1351146-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9989653809809362
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1351146-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9325551813947633
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1514320-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9877758066120408
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1514320-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5960501873943282
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1514320-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8170560017523872
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1602272-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9981454640403801
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1602272-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996396130677092
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1602272-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9934236077992445
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1602272-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.618460528270965
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1714895-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999065975971743
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1714895-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997647697736528
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1714895-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.30508648826217466
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1768914-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9796183429668995
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1768914-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9770385426154212
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1768914-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9704006497566654
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1768914-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.998474895602483
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1768914-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9856908365206151
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-1768914-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.996796411327245
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2034017-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.4935265150066973
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2034017-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999628177790041
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2034017-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996153257102737
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2034017-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999348297455227
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2034017-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9985664951672554
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2034017-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9941662553139972
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2131702-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9943448584568877
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2131702-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995326055787108
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2131702-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9944477219202033
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-227418-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9968244218872427
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-227418-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.993414129116223
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-227418-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9935215283863145
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-227418-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996999251210773
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2429208-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.992679705461366
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2429208-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9967211882477691
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2429208-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5119870735650647
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2772694-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9953367744339288
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-2772694-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996266479464135
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3060399-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9940544008372936
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3060399-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.997780346801358
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3060399-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9934223974157452
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3060399-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9784663090655025
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3105669-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9979984777381361
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3105669-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9762338208966395
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3105669-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996502160283052
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3105669-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9909759069775311
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3105669-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9334907081102809
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3127602-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.958726172633623
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3127602-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9782948416856886
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3436159-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9939712913765212
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3436159-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9967612616871747
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3436159-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.409036502610057
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3865791-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9946370524061198
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3865791-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9837160706758913
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3865791-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9914654674598946
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3865791-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.998869428955989
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3865791-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9978200697086761
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3961172-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9807073857679895
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3961172-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9981009594059287
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3961172-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9779009927925785
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-3961172-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5809428544559254
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4006043-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9941812032278443
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4006043-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9891733416416445
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4006043-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9987614088562389
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4059589-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9988721775601832
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4059589-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9965759303400042
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4059589-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9965759303400042
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4059589-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9420609255211916
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-40699-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9878898417522148
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-40699-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5051966161257193
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4070745-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9971186258786435
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4130801-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.852838429202703
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4130801-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996436213572637
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4130801-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8761658047523171
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4161430-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997406129720136
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4161430-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999939346860884
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4161430-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993989241929812
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4167818-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995966911068505
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4167818-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9504655474477173
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4167818-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9991658551488299
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4269599-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9180589033713997
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4269599-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9998017821542353
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4269599-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9812480431149659
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4269599-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9935353234721589
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4269599-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8639805725013259
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4269599-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9982403609739121
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4269599-0007.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994333376713952
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4269599-0008.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999085842401563
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4269599-0009.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5035775661874726
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4322734-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999700234820915
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4322734-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995838993903973
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4345731-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9998621970944168
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4345731-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9958181035660585
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4345731-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9970808683452671
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4345731-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5035775661874726
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-4376803-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9470008849662949
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5141192-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9857602019534273
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5141192-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999939542275422
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5141192-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.27209695291704833
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5155977-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993188572959535
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5155977-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9947960520463174
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5155977-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9942926331368303
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5155977-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9985139073220717
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5155977-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999750582799447
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-518323-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999568277764013
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-518323-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9802623110211265
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-518323-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9566973054544398
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5240350-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9982551489735588
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5240350-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9985591325969863
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5240350-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9998947155535532
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5240350-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8250708137547713
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5421131-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996232795159742
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5421131-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9976145240847647
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5451280-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999866863822608
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5451280-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7199699292502154
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-545831-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9992555543786554
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5489623-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9928992563530139
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5489623-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996844051689061
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5489623-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7416126762596313
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5660566-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999334003873722
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5660566-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995418121394765
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5660566-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9945140499799856
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5660566-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9973957410939032
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5660566-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9987907533371809
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5660566-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9911801632189188
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5660566-0007.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994965219107351
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5660566-0008.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7660620023321344
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5754167-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.989987515353521
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5754167-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7845690453745594
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-5757955-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.17106837590867321
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6105615-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9983260167198912
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6105615-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.998882289564755
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6168941-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997594048465168
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6168941-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9890755850251928
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6402155-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997485341703131
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6402155-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7754125349997081
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6402155-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994646383161038
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6402155-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.31044148592666715
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6523466-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9986229960889005
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6523466-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997922380807509
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6806075-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8314177741890567
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6806075-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9982370334630376
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6806075-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995335897280567
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6806075-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5152824866923542
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6806075-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.6961168391536314
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6833685-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9970267794460086
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6833685-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9904426907338169
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6833685-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.3871323725054439
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6950003-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5035775661874726
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6950003-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9590947101802051
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6950003-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9950441783458212
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6950003-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9998781880028338
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6950003-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9954588121294761
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6950003-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9949295964916074
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6950003-0007.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9981460884256486
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-6987875-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.933471888074465
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7051071-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999869739274508
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7051071-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999951311471727
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7098043-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9992844439000129
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7136666-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9985213591401644
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7257218-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5035775661874726
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7257218-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9979559322224988
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7257218-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9976073631973446
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7257218-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.997968500121552
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7257218-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5126582152714428
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7346761-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.999825925861841
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7720431-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9931785251035173
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7720431-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9950763190585836
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7720431-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.996875957155867
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-7720431-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.4330470024574012
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8046863-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9961234657490103
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8046863-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999596713351706
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8190872-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8919667560373373
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8190872-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9900797708487638
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8190872-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9992470189413512
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8190872-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993263205856849
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8190872-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9859486968888248
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-821920-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995241032738775
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-821920-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.4831951845738851
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8285922-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9950437115693201
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8285922-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9909351698610604
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8285922-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9929113538343576
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8285922-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9975013303668894
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8285922-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9975013303668894
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-840611-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.997461766835255
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-840611-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994898621999505
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-840611-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999704443164803
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-840611-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996244803362176
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-840611-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7221940202056395
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-846757-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9968734702696905
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-846757-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9998838732155345
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8563288-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994610826208801
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8837254-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9988789888022563
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8846908-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9992957262677246
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8846908-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9910289222697741
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8846908-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5426667157939747
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8983865-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9998863855557135
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8983865-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997680346342486
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8983865-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9998018357278715
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8983865-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.992830198670639
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-8983865-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.6001416416090852
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9050379-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.989667433156452
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9060950-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9986770981967484
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9060950-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9761852435149014
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9156003-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995471196045291
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9156003-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999194799066883
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9156003-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997001646240349
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9156003-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993301639024739
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9156003-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.30594955061415646
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9182718-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9961897716645169
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9242636-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.4863539864966656
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9242636-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9929637201302026
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9242636-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999904053064375
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9242636-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9832877444642535
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9242636-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999877266646012
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9242636-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.452059350854646
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-932292-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9950114632117358
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-932292-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9997803250612494
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-932292-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9998102845693502
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9364007-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999282922149024
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9364007-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9990507272686048
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9364007-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.29195800483215084
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-949117-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994720587038202
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9503383-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995323785241751
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9673743-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995540631368922
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9673743-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996781916725513
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9673743-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9922975660546463
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9764136-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9778325637665531
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9857270-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9999745721452173
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9857270-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9975287672190583
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9857270-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7996756496842428
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9857270-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.31164906749317395
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9913238-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9987030775384518
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9913238-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.983520496586147
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9913238-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9657993350335129
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1056329-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9029971593569295
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1142234-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.6956003937813624
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1207991-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9747354610057298
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-131422-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.952624611665074
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1352878-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995527306226653
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1352878-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994907511538994
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1352878-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8273257775620878
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1352878-0004.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9952481884122429
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1352878-0005.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9992433472503917
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1352878-0006.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9991257801887512
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1352878-0007.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9983592867000359
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1423083-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9988203437487846
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1463523-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9881649236744016
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1517175-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9550001986218031
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1540910-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9902016364793278
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1540910-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9896148348947803
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-1795675-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9963564165873553
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-2040583-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9971853984294947
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-2040583-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993047448665303
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-2061674-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9929741951085714
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-2102459-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9824037254725297
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-2344316-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9890229080647098
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-2352342-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9973798621802573
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-2352342-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.998777586491376
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-2470676-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9958234219909085
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-258000-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9082728730009018
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-262135-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9417800974374373
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-280992-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9639794011101412
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-2831144-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9337265099106995
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-2914571-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9977081251383203
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3008952-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9944785093193932
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-30184-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5219862959553183
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3028666-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9898283452707017
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3039338-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9930898493560543
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3119878-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.590722016784309
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3174339-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8646593040825803
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3225323-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9938683316788399
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3239718-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9758734372045894
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3239718-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9984800915538793
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3426682-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9945425205068542
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3439481-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9848041665093916
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3522052-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9940414511807156
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3565661-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.84447930754305
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3635859-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9433563085541218
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3698459-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9010886521380933
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3746644-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9800151585316698
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3768496-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9691475520390852
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3814177-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9449411249055966
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3828598-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9947416737041495
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3828598-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9979127871692814
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3835085-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9816859989583031
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3835085-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9975044202940893
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3838349-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9983272733289333
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3906469-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.5498699576400361
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-3915569-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.98588339905186
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4009702-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9888743718900206
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4033321-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.99972486198085
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4190128-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.994341267019794
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4275213-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7345342684566623
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4276353-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.997012291053177
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-432590-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9985923878722557
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4355102-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9936102049044002
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4360206-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9966706727567515
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4405644-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9559352159088326
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-447130-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996429949917757
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4533169-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8162120644605894
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4655012-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9680358471705914
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4664910-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996153520210227
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4810299-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.878546206416458
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4810299-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9908273876241344
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-48197-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9993920036022208
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4840553-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9928442916766663
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4852350-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9749879833645084
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-490019-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994406531845116
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-490019-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994406531845116
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-4911614-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9515828724430084
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5069337-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9695521009389901
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5180281-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9900469920421163
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5271295-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9072062868044121
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5366974-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9692456861047722
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5391810-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9994918196334167
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5508710-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9570982952629105
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5660566-0009.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9940367425902835
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5708148-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9932923430091466
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5708148-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9877919320400034
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5741681-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9715180401187615
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5813112-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9413762522417102
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5864551-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9959407569880987
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5880719-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9819035220350891
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-5927483-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9953312502773712
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6076183-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8482478023017853
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6076183-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.6756302797403237
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6079770-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9143229718370065
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6138035-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9978511386493079
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6207158-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9955222973156591
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6249454-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9974992318185514
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6249454-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9903159326174356
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6249454-0003.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9742858864997389
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6682185-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9500706121932766
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6733553-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9964288767729759
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6812602-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9904787780262664
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6828836-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7357545911906386
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6831597-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9824296636184396
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6840343-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9551737023848484
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6844248-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7363233991528467
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-6844248-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9612959238959358
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7014527-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9699627509080881
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7014527-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9951871399616402
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7318158-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8877536219090787
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7337079-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.988605294389708
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7385225-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.7063010476424256
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7441806-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9955907188077975
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7441806-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9885774511850087
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7477247-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.3922046950491694
Prediction is: NORMAL
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7589381-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9784764717258372
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7638941-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8927409078288708
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7688430-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9370492792339112
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7788460-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9990627436484621
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7822291-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.993726482200653
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7909775-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9827694294382227
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-7979-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9989240944561871
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-8106836-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9891362492737271
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-820420-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9033595934109355
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-8314555-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.6810705650894825
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-8377377-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9958713788391991
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-8385372-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9979548055317571
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-842024-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9663917290707738
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-8427429-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9968257604659837
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-8435135-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9028324436932745
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-8507816-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9964527585380804
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-8544560-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9979150761965347
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-8574414-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9949068935014239
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-874167-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9901389540607524
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-8903181-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9827040711939335
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9018258-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9940078594739867
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9039335-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9703134409306517
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-904779-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9763711505723708
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9240664-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.8172228608961446
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9325276-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.987209766881511
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9342284-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9823872258680323
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9476607-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9990661511590725
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9564688-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9996939252589763
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9564688-0002.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.994890511999061
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9584831-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9917859201522394
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9671740-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9905110437099386
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9783315-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9951915721314664
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9890836-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9983520627700052
Prediction is: PNEUMONIA
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/VIRUS-9968655-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.894028663904279
Prediction is: PNEUMONIA

Visualizing uncalibrated probability diagram

Using ml_insight library:

In [34]:
import ml_insights as mli
uncal_prob_test = np.array(uncalibrated_Pneumonia_probabilities_test)
y_test = np.array(test_y)
plt.figure(figsize=(10,5))
mli.plot_reliability_diagram(y_test, uncal_prob_test, show_histogram=True);

Just a side note:

ml_insights library sometime has issues when importing on AWS. We could do the same plots using scikit-learn. Example shown below:

In [49]:
plt.figure(figsize=(10,5))
plt.hist(x=uncalibrated_Pneumonia_probabilities_test, bins=10, color='#0504aa', alpha=0.5, rwidth=0.75);

plt.figure(figsize=(10,5))
prob_true, prob_pred = calibration_curve(test_y, uncalibrated_Pneumonia_probabilities_test, n_bins=10)
disp = CalibrationDisplay(prob_true, prob_pred, uncalibrated_Pneumonia_probabilities_test);
disp.plot()
Out[49]:
<sklearn.calibration.CalibrationDisplay at 0x7f5533a79f40>
<Figure size 720x360 with 0 Axes>
In [51]:
p1 = np.array(prob_true)
p2 = np.array(prob_pred)
error = p1 - p2
error_square = np.square(error)
error_sum_square = np.sum(error_square)
print(f"Uncalibrated probability Sum of Squares: {error_sum_square}")
Uncalibrated probability Sum of Squares: 0.29199751521036654

We note from the result that the uncalibrated probabilities are highly over-estimated as we expected from visualizing the same on the smaller calibration dataset.

From histogram, we note that we dont have many points in probability regions around 0.6 to 0.8 and this results in noisy points in the calibration diagram. This is an issue with this dataset as even looking at the entire dataset, there are not enough points. However, this does not much affect overal performance and we can focus on regions with more datapoints.

We also defined a metric as Sum of Squared Error that illustrated the difference between predicted and actual probabilities.

Visualizing uncalibrated probability diagram

Now, we send the uncalibrated probabilities to the calibration model to get the calibrated probabilties and then visualize the output.

Using scikit-learn:

In [52]:
calibrated_Pneumonia_probabilities_test = betaCalibration.predict(uncalibrated_Pneumonia_probabilities_test)
plt.figure(figsize=(10,5))
prob_true, prob_pred = calibration_curve(test_y, calibrated_Pneumonia_probabilities_test, n_bins=10)
disp = CalibrationDisplay(prob_true, prob_pred, calibrated_Pneumonia_probabilities_test)
disp.plot()
Out[52]:
<sklearn.calibration.CalibrationDisplay at 0x7f553037b520>
<Figure size 720x360 with 0 Axes>

Using ml_insight library:

In [53]:
cal_prob_test = np.array(calibrated_Pneumonia_probabilities_test)

mli.plot_reliability_diagram(y_test, cal_prob_test);
In [54]:
p1 = np.array(prob_true)
p2 = np.array(prob_pred)
error = p1 - p2
error_square = np.square(error)
error_sum_square = np.sum(error_square)
print(f"Calibrated probability Sum of Squares: {error_sum_square}")
Calibrated probability Sum of Squares: 0.22386608494114665

We can notice much improvement in the diagram. The points from the model are generally closer to the actual probability.

Our Sum of Squared Error metric is also improved. Although the difference seem small but it is a significant improvement.

Applying calibration to the endpoint in production

As noted earlier, we could do the probability calibration inside the script and deploy that together with the CNN model to the endpoint. But we did this in the notebook instance for illustration.

This could still be a good choice becuase the calibration model can later on be applied to the logits received from the endpoint. Another option is that a Lambda function interfacing with the endpoint can apply the calibration to the endpoint result and return the calibrated probabilities to the client.

For illustration, we randomly select 5 X-ray images from the test dataset, call the endpoint, apply the calibration to the response from the model, and report both calibrated and uncalibrated probabilities. For visualization, the image used for this inference is also shown.

In [57]:
from sagemaker.serializers import IdentitySerializer

predictor.serializer = IdentitySerializer("image/jpeg")
jpeg_serializer = IdentitySerializer("image/jpeg")

session = boto3.Session()
s3_client = session.client('s3')
s3 = boto3.resource('s3')   


test_len = len(test_data)
sample_index = random.sample(range(test_len), 5)

sample_images = [test_data[index] for index in sample_index]
sample_labels = [test_y[index] for index in sample_index]


from PIL import Image
import numpy as np
bucket_name = bucket
bucket_ = s3.Bucket(bucket_name)
uncalibrated_Pneumonia_probabilities_test = []
for file, label in zip(sample_images, sample_labels):
    print("--------------------------")
    print("Image: " + file)
    print("Label: " + str(label))
    object = bucket_.Object(file)
    response = object.get()
    file_stream = response['Body']
    img = Image.open(file_stream).convert('RGB')
    display(img)
    count += 1
    
    s3_object = s3_client.get_object(Bucket = bucket_name, Key = file)
    body = s3_object['Body']
    content = body.read()
    response = predictor.predict(content, initial_args={'ContentType':'image/jpeg'})
    probs = softmax(response[0])
    uncalibrated_Pneumonia_probability = probs[1]
    print("Uncalibrated Pneumonia Probability: " + str(uncalibrated_Pneumonia_probability))
    calibrated_Pneumonia_probability = betaCalibration.predict([uncalibrated_Pneumonia_probability])
    print("Calibrated Pneumonia Probability: " + str(calibrated_Pneumonia_probability[0]))
    prediction = np.argmax(response[0]) + 1 
    prediction = 'NORMAL' if np.argmax(response[0]) == 0 else 'PNEUMONIA' 
    
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-6395296-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.7172762632109436
Calibrated Pneumonia Probability: 0.727611282798618
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-7165123-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.9507210344862049
Calibrated Pneumonia Probability: 0.9396063458887806
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-630880-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8712555385954821
Calibrated Pneumonia Probability: 0.8828038394175102
--------------------------
Image: chestCRXImages/chest_xray/test/NORMAL/NORMAL-4315572-0001.jpeg
Label: 0
Uncalibrated Pneumonia Probability: 0.8219498789855713
Calibrated Pneumonia Probability: 0.8416779295391046
--------------------------
Image: chestCRXImages/chest_xray/test/PNEUMONIA/BACTERIA-9673743-0001.jpeg
Label: 1
Uncalibrated Pneumonia Probability: 0.9995540631368922
Calibrated Pneumonia Probability: 0.9928922463476831
In [ ]:
 

References:

  • Medical imaging in personalized medicine: a white paper of the research committee of the European Society of Radiology (ESR). Insights Imaging 6, 141–155 (2015). https://doi.org/10.1007/s13244-015-0394-0
  • E. Sogancioglu et. al., Deep Learning for Chest X-ray Analysis: A Survey, Medical Image Analysis, arXiv:2103.08700
  • Raoof, S., Feigin, D., Sung, A., Raoof, S., Irugulpati, L., Rosenow, E.C., 2012. Interpretation of plain chest roentgenogram. Chest 141, 545–558. doi:10. 1378/chest.10-1302 Balabanova, Y., Coker, R., Fedorin, I., Zakharova, S., Plavinskij, S., Krukov, N., Atun, R., Drobniewski, F., 2005. Variability in interpretation of chest radiographs among russian clinicians and implications for screening programmes: observational study. BMJ 331, 379–382. doi:10.1136/bmj. 331.7513.379
  • Kermany DS, et. al., Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning. Cell. 2018 Feb 22;172(5):1122-1131.e9. doi: 10.1016/j.cell.2018.02.010. PMID: 29474911.
In [ ]:
 
In [ ]: